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Generating a video stream object in a data editing system 



(54) 

(57) A desired video stream object is created from multimedia data in a data editing system. Primitive objects 
having a rectangular object, a position object and an interval object are extracted from the multimedia data. 
Media objects are then created by extracting spatial and temporal features based on the multimedia data and 
the primitive objects. Temporal relations among the media objects are determined wherein each temporal 
relation represents a playback start time and a playback length of a corresponding media object and a time 
scale object is created to adjust the playback length of the corresponding medium based on the temporal 
relation thereof. A spatial scale object is created to control the size of a region of a corresponding medium 
being represented on a screen of the system based on the spatial relation thereof. The desired video stream 
object is created based on the temporal relations and the temporal scale object and the spatial relations and 
the spatial scale object. 
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METHOD AND APPARATUS FOR GENERATING A VIDEO STREAM OBJECT 

IN A DATA EDITING SYSTEM 

j5 The present invention relates to a data editing system; 

and, more particularly, to a method and apparatus for 
generating a desired video stream object from multimedia data 
in the data editing system. 

10 

In recent years, editing technology to manipulate 
multimedia documents that include picture, audio, text, and 
other modifiable objects has achieved a significant progress. 
Among the various editing techniques, there are in 

15 particular two kinds of editing techniques that have been 
widely used: a video algebraic editing technique and a 
hierarchical multimedia document data editing technique. The 
video algebraic editing technique carries out such functions 
as creation, output, explanation, composition and the like to 

2 0 manipulate video streams of motion pictures. The creation 
function serves to create an input video for editing and the 
■ output function is to simultaneously output the video streams 
by employing several preset windows. And the explanation 
function adds, to each video stream, additional information 

25 for explaining it; and the composition function processes one 
or more video stream data by using preset operators, e.g., 
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continuation, summation, crossing, subtraction, repeating, 
extension, limitation and the like. Using the above-mentioned ' 
functions, it is possible to edit a desired video stream and 
change its contents by properly composing more than one video 
stream data. However, the prior art algebraic editing 
technique does not provide an editing scheme for media, e.g., 
still picture., text, sound and. animation, which are not the 
video stream data of the motion picture. 

The hierarchical multimedia document data editing 
technique has been developed to solve the above problem in the 
algebraic editing technique. The hierarchical multimedia 
document data editing technique employs four hierarchical - 
layers, i.e., a definition layer, a manipulation layer, a 
presentation layer and a control layer. Specifically, the 
definition layer deals with the definition of a class to which 
an object of multimedia document data belongs; and the 
manipulation layer handles temporal features between objects 
in the multimedia document data'. An object having the 
temporal feature refers to an event, wherein operators such 
as concatenation, overlay, extension and insertion are used. 
Further, at the presentation layer/ user interface components 
are defined, wherein the components may include, a spatial 
' layout between the- objects, output type, window and icon. At 
the control layer, an interaction between a user and each of 
25 an input and an output sections is defined. 

However, in such a hierarchical editing technique, the 
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multimedia document data is processed, using the four layers, 
without taking into account temporal and spatial relations 
among media objects on the basis of motion picture, .thus 
making it difficult to meet the requirement for 
synchronization between the motion picture and each of the 
media objects. 

It is, therefore, a primary object, of ' the present 
invention to provide a method and apparatus which is capable 
of creating a desired video stream object with an accurate 
synchronization between media objects therein by taking into 
account temporal and spatial relations between a motion 
picture and the media objects. - 

* In accordance with one aspect of the present invention, 
there is provided a method for generating a desired video 
stream object from multimedia data having the form of a video 
stream in a data editing system, comprising the steps of: 

(a) extracting primitive objects having a rectangular 
object, a position object and an interval object by using head 
information in the multimedia data; 

(b) creating media objects based on the multimedia data 
and the primitive objects, wherein the media objects are 
classified into temporal and spatial objects; 

(c) _ determining temporal relations among the media 
objects, wherein each temporal relation represents a playback 



start time and a playback length of a corresponding media 
object on the basis of a motion picture object out of the 
media objects and creating a temporal scale object to adjust 
the playback length of the corresponding media object based 
on the temporal relation thereof; 

(d) determining spatial relations among the media objects 
wherein each spatial relation represents a position and a size 
of a region of a' corresponding media object being represented 
on a screen of the system and creating a spatial scale object 
to control the size of the region based on the spatial 

relation thereof; and 

(e) creating the desired video stream object based on the 
temporal relations and the temporal scale object and the 
spatial relations and the spatial scale object. 

in accordance with another, aspect of the present 
invention, there is provided an apparatus" for generating a ' 
desired video stream object from multimedia data having the 
form of a video stream in a data editing system, the apparatus 
comprising: 

means for extracting primitive objects having a 
rectangular object, a position object and an interval object 
by using head information in the multimedia data .and creating 
media objects based on the multimedia data and the primitive 
objects, wherein the media objects are classified into 
25 temporal and spatial objects; 

means for determining temporal relations and spatial 
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relations among the media objects wherein each temporal 
relation represents a playback start time and a playback 
length of a corresponding media object on the basis of a 
motion J picture object out of the media objects and each 
spatial relation depicts a position and a size of a region of 
a corresponding media object being displayed on a screen of 
the system; 

means for creating a time scale object to adjust the 
playback length of the corresponding media object based "on the 
temporal relation thereof and creating a spatial scale object 
to control the size .of the region of the corresponding media 
object based on the spatial relation thereof; and 

means for generating the desired video stream object 
based on the temporal .and the spatial relations and the 
temporal and the spatial scale objects. 

The above and other objects and features of the present 
invention will become apparent from the following description- 
of preferred embodiments given in conjunction with the 
following accompanying drawings, in which: 

■ Fig. 1 shows a block diagram of a video stream object 
generation- apparatus in a data editing system in accordance 
with the present . invention; 

Fig. 2 presents an exemplary drawing for describing a 
hierarchical structure of a media object in accordance with 



the invention; and 

Figs. 3 and 4 offer illustrative drawings for explaining 
temporal and spatial relations . between media objects in 
accordance with the invention, respectively. ■ 



' Referring to Fig. 1, there is shown a block diagram of 
a video stream object generation apparatus 100 for generating 
a desired video stream object from multimedia data in a data 
editing system in accordance with the present invention. As 
will be fully described below, "the present invention employs 
an objected-oriented multimedia data editing scheme which 
takes account of temporal and spatial relations between media 
'objects. The video stream object generation apparatus 100 of 
the invention comprises a media object generation device 10, 
a synchronization (SYNC) object generator 20, a spatial- 
composition object generator 30 and a video stream object 
generator 40. 

Multimedia data is first applied to the media object 
generation device 10 in the form of a video stream, wherein 
t „e multimedia data is non- structured data consisting of medxa 
such as a .motion picture, a still picture, a text, a sound and 
the like. The multimedia data may be stored in advance in a 
database (not shown, associated with the device 10 or a 
storage unit similar to the database.^ At the media object 
generation device 10, media objects are created by using the 



multimedia data. . For the creation of the media objects, first 
of all, primitive objects such as a rectangular object, an 
interval object and a position object are set at 'a primitive 
object generator included in the media object generation 
5 device 10. 

Among the primitive objects, the position object and the 
rectangular object depict a position and a size of a region 
where a spatial media, as will be discussed below, is being 
represented on a screen (not shown) associated with the 

1G inventive apparatus 100, respectively. To be more specific, 
the position of the region is' expressed as a coordinate (X, 
Y) on a horizontal and a vertical axes, whereas the size 
thereof is defined as a function of a horizontal and a 
vertical lengths in units of pixel on the horizontal and the 

15 vertical axes. In the coordinate, a reference point is a 
point positioned at an upper left comer of the region. And 
the interval object stands for a playback interval of a 
temporal media, as will be discussed below. The playback 
interval is expressed as a function of a playback start point 

2 0 and a playback end point of the temporal media, wherein units 
-of the start and the " end points are second. . The horizontal 
and the' vertical lengths and the start and the end points may 
be obtained by extracting corresponding data- in head 
information of the multimedia data. 

25 For the creation of the media objects, a specific or 

interest region object is further set by the user, wherein the 
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specific or interest region stands for. a region where the user 
regards it as an interest or importance region within a region 
of media being represented on the screen. The specif ic region 
of a temporal object is expressed as a function of an interval 
object which is newly set by the user, while the specific 
region of a spatial object is defined as a function of a 
rectangular object and a position object which are also newly 
set by the user. These temporal and spatial specific region 
objects are used for the user to adjust a playback length and 
a representation region of respective corresponding media, 
respectively, to obtain a desired video stream object. Once 
the specific region objects are created with respect to each 
me dia object, the remaining region excepting the specific 
regions is decided to be an auxiliary region object. 

After creating the primitive objects and the specific, 
region objects, the media objects are now created by using the 
objects at a media object generator ' included in the media 
object generation device 10. A media object 
hierarchical structure as illustrated in Fig. 2. In other 
words, the media object is classified into two types of 
objects, a temporal object and a spatial object. The temporal 
object contains temporal features such -as a playback length 
' and.a playback speed of each temporal media and the spat.al 
ohject contains spatial features such as information related 
to a region or portion *f each spatial media being displayed 
on the screen. As shown ' in Fig. 2, it is known that the 
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motion picture object contains attributes of both of the 
temporal object and the spatial object, while the still 
picture object and the text object contain attributes of only 
the spatial object and the sound object contains attributes 
5 of only the temporal object. Further, it is possible to 
create the text object by applying contents, font, color, 
rectangular, etc. of words therein 'to a text generator (not 
shown) contained in the media object generator. Data on each 
of' the primitive objects, the specific region objects and the 

10 media objects/ created at the media object generation device 
10, is then provided to the SYNC object generator 2 0 and also 
to the spatial composition object generator -30. 

At the SYNC object generator 20, temporal relations 
between the media objects are first determined based on the 

15 .motion picture object. Each-of the temporal relations refers 
to a function of a playback start frame and a playback length 
of a corresponding one of the media objects on the basis of 
the motion picture object. Fig. 3 shows the temporal 
relations between the motion picture object and each of the 

2 0 text, the still picture and the sound objects, wherein the 
horizontal axis represents the number of frames and the 
- vertical axis . indicates the type of object. In Fig. 3, for 
instance , ' the playback start frames of the text object are a 
1st one and an 80th one, respectively, and the playback 

25 lengths thereof are 10 frames and 20 frames, respectively. 

After determining the temporal relations, in a preferred 
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embodiment of the invention," the playback length of each 
temporal media object is adjusted by using a temporal scale 
object which will fully be described hereinafter. The time 
scale object is used to adaptively adjust the temporal feature 
of each media object to satisfy the requirements of 
specification defined in the motion picture object. The time 
scale object is expressed as a function. of a name of a media 
object and a time scale adjustment technique. There are two 
types of temporal scale adjustment techniques: One is an 
origin point based adjustment technique and the other is a 
specific or interest region adjustment technique. The origin 
point based adjustment technique depends on a start point of 
each media, object, whereas the specific region adjustment 
technique is based on an interest or importance 'region 
determined by the user within the region of said each media 
object. By composing the temporal relations and the temporal 
scale object decided above, a SYNC object for each media, 
object is now created at the SYNC object generator 20. Data, 
on each of the SYNC objects' created at the SYNC object 
generator 20 is then transferred to the video stre.,:. object 
generator 40 . 

' m the meantime, at the spatial composition object 
generator 30, spatial composition objects are created. To 
create the spatial composition objects , the spatial relations 
between the motion picture object and each of the spatial 
m edia objects are first determined and then spatial scale 



object for each media object is created based on the spatial 
relations. Each spatial relation refers to a function of a 
position and a size of a region of each media object on the 
motion picture object being represented on the screen. As 
5 described above, the position of the region is • expressed as 
a coordinate (X, Y) with respect to a coordinate origin 
positioned at an upper left corner of the region and the size 
thereof represents : a horizontal and a vertical lengths/ in 
units of pixel of the region on a horizontal and a vertical 

10 axes. Fig. 4 depicts an exemplary drawing for explaining 
spatial ■ relations between the media objects on the .basis of 
the motion picture object in accordance with the invention. 
In Fig. 4, for example', the. position of the text object is 
given by a coordinate (10, 280) and as for the size thereof, 

15 the horizontal length is 280 and the vertical length is 10. 
Of course, as discussed, unit of the length is pixel. 

For the creation of the spatial composition objects, in 
accordance with the 'invention, there are further set a 
priority representing the playback order of the media objects 

2 0 and a movement object depicting a movement direction and a 
movement speed of a region of each media object being 
displayed on the screen. Using the movement object, it is 
possible. to adjust the position of each media object during 
the playback thereof. In accordance with a preferred 

25 embodiment of the invention, the movement direction is set to 
be one along the left, right, upper and lower; and the 
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movement speed is decided as the number of pixels per a frame 
in the motion picture object. 

As in the SYNC object generator 20, the size of the 
region of each media" object being displayed on the screen can 
also be adjusted by using a spatial scale object which will 
be described in detail hereinafter. The spatial scale object 
is used to adaptively adjust the spatial feature of each media 
-object to satisfy the requirements of the specification 
define* in the motion picture object . The spatial scale 
object is defined as a function of a position and a size of 
a region of each media object being displayed on the screen. 
Also the spatial scale object may employ either one of the 
origin point based adjustment technique and the specific 
region adjustment technique fully described above. Of course, 
it is evident that the origin point based adjustment technique 
depends on a point .positioned at an upper left corner of the 
region. Using the spatial" relations and the spatial scale 
object, the priority, and the movement object, a spatial 
condition object for each media object is now created at the 
spatial composition object generator 30. Data on each of the 
spatial, composition objects created is then provided to the 
video stream object generator 40. 

At the video stream object generator 40, a desired video 
stream object is created by composing the temporal relations 
and the temporal scale object from the SYNC object generator 
20 and the spatial relations and the spatial scale object from 
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the spatial composition object generator 30. The created 
video stream object may be a desired edited one or may be used 
. as a reference object to create a different video stream 
object. As discussed above, the present invention employs an 
5 objected-oriented multimedia data editing scheme which takes 
account of the temporal and the spatial relations between the 
media objects on the basis of the motion picture .object; and, 
therefore, it is possible to create all desired objects and 
to accomplish an accurate synchronization between a motion 

10 picture media and other media in the object. 

While the present invention has been described with 
respect to certain preferred embodiments only, other 
modifications and variations may be made without departing 
from the scope of the present invention as set forth in the 

15 following claims . 
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Claims : . 
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1. A method for generating a desired video stream object 
from multimedia data having, the form of a video stream in a 
data editing system, comprising the steps of: 

(a) extracting primitive objects having a rectangular 
object, a position object and an interval object by using head 
information in the multimedia data;. 

(b) creating media objects based on the multimedia data 
and the primitive objects, wherein the media objects are 
classified into temporal and spatial objects; 

(c) determining, temporal relations among the media 
objects, wherein each temporal relation represents a playback 
start time and a playback length of a corresponding media 
object on the basis of a motion picture object out of the 
media objects and creating a temporal scale object to adjust 
the playback length of the corresponding media object based 
on the temporal relation thereof; 

(d) determining spatial relations among the media objects 
wherein each spatial relation represents a position and a size 

.of a region of a corresponding media object being represented 
. on a screen of the system and creating a spatial scale object 
to control the size of the region based on the. spatial 

relation thereof; and 

(e) creating the desired video stream object based on the 
temporal relations and the temporal scale object and the 
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spatial relations and the spatial scale object. 

2. The method of claim l, wherein the interval object 
represents a playback interval of the corresponding media 
object and the position object - and the rectangular object 
depict the position and the size of the region of the 
corresponding media object being represented on the screen, 
respectively, wherein the playback interval is defined as a 
function of a playback, start point, and a playback end point 
of the corresponding media object, the position of the region 
is expressed as a coordinate with respect to a coordinate 
origin on a horizontal and a vertical axes, and the size of 
the region is defined as a function of a horizontal and a 
vertical lengths in units of pixel on the horizontal and the 
vertical axes. 

3. The method of claim 2, wherein the temporal and the 
spatial scale objects selectively employ one of an origin 
point based adjustment technique and a specific region 
adjustment technique to adjust the playback length of .the 
corresponding media object and the 1 - position and the size of 
the region of the corresponding media object, respectively, 
to synchronize it with the motion picture object, wherein the 
origin point based adjustment technique depends on, for the 
temporal scale object, a playback start point of the 
corresponding media object and, for the spatial scale object, 



a point positioned at an upper left corner of the region of 
the corresponding media object; and the specific region 
adjustment technique is based on an interest region set by the 
system user, the interest region being defined as a region 
5 where the user regards it as an interest region within the 
region of the corresponding media object being represented on 
the screen. 

4. The method of claim 3, wherein the step (d) includes a 
10 step of setting a sequential order of the media objects to be 

represented on the screen. 

5. The method of claim 4, wherein the step (d) further 
includes a step of creating a movement object to change the 

15 position of each media object during the playback thereof, 
wherein the movement object depicts a movement direction and 
a movement speed of said each media object to be represented 
on the screen. 

20 " 6. An apparatus for generating a desired video stream object 
from multimedia data having the form of a video stream in a 
data editing system, the apparatus comprising: 

means for extracting primitive objects having a 
rectangular object, a position object and an interval object 

' 2 5 by using head information in the multimedia data and creating 
media objects based on the multimedia data and the primitive 
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objects, wherein the media objects are classified into 
temporal and spatial objects; 

means for determining temporal relations and spatial 
relations among the media objects wherein each temporal 
•relation represents a playback start time and a playback 
length of a corresponding media object on the basis of a 
motion picture object out of the media objects and each 
spatial relation depicts a position and a size of a region of 
a corresponding media object being displayed on a screen of 
the system; * 

means for creating a time scale object to adjust the 
playback length of the corresponding media object based on the 
temporal relation thereof and creating a spatial scale object 
to control the size of the region of the corresponding media 
object based on the spatial relation thereof ; and 

means for generating the desired video stream object 
based on the temporal and the spatial, relations . and the 
temporal and the spatial scale objects. 

7. The apparatus of claim 6, wherein the interval object 
represents a playback interval of the corresponding media 
object and the position object and the rectangular object 
depict the position and the size of the region of the 
corresponding media object being represented on the screen, 
respectively, wherein the playback interval is defined as a 
function of a playback start point and a playback end point 



of ,ae corresponding media object, the position of the region 
is expressed as a coordinate with respect to a coordinate 
origin on a horizontal and a vertical axes, and the size 
thereof is defined as a function of a horizontal and a 
vertical lengths in units of pixel on the horizontal and the 
vertical axes. 

8. The apparatus of claim 6, wherein the temporal and the 
spatial . scale objects selectively employ one ' of an origin 
point based adjustment technique and a specific region 
adjustment technique to adjust the playback length of the 
corresponding media object and the position and the size of 
the region thereof, respectively, to synchronize it with the 
mo tion picture object, wherein the origin point based 
adjustment technique depends on, for the temporal scale 
object, a playback start point of the corresponding media 
object and, for the spatial scale object, a point positioned 
at an upper left corner of the region of the corresponding 
media object; and the specific region adjustment technique is 
based on an interest region set by the system user, the 
interest region being defined as a region where the user, 
regards it as an interest region within the region of the 
corresponding media object being represented on the screen. 

9. The apparatus of claim 8, wherein the temporal and 
spatial relation determining means includes means for setting 
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a sequential order of the spatial media objects to be 
represented on the screen. 

10. The apparatus of claim 9, wherein the temporal and 
spatial relation determining means further includes means for 
creating a movement object to change the position of each 
media object during the playback thereof, wherein the movement 
object represents a movement direction and a movement speed 
of said each media object to be represented on the screen. 

11. A video stream object generation apparatus constructed 
and arranged substantially as herein described with reference 
to or as shown in Figs. 1-4 of the- accompanying drawings. 
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